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FIRE  Experiment  Planning  and  Reporting  Structure 
Guidelines  -  June  2007 


Section  1.  Introduction 


The  experimentation  structure  utilizes  three  planning  forms  and  two  reporting  forms.  Figures  1 
and  2  show  their  content.  Brief  content  descriptions  accompany  the  figures.  Complete 
descriptions  are  in  following  sections. 
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Figure  1.  Schematic  of  FIRE  planning  forms. 


Objective  planning  specifies  what  is  to  be  learned  and  how  that  will  be  done.  What  is  to  be 
learned  is  specified  by  an  Objective  and  associated  Objective-Questions.  Attributes  to  be 
addressed  are  contained  in  the  Objective-Question;  their  Measures  and  Data  sources  for  those 
measures  are  also  specified.  Some  Attributes  will  be  addressed  with  survey  questions. 


Data  planning  specifies  the  events  and  situations  that  are  needed  to  produce  the  required  data. 
Also  included  are  data  collection  procedures  and  collector  instructions  that  direct  their  activities. 
The  data  planning  procedure  and  form  are  called  ThreadEx. 


Analysis  planning  specifies  the  procedures  for  producing  the  various  quantitative  measures  and 
how  subjective  survey  information  is  to  be  processed.  Also  included  are  procedures  for 
producing  the  answers  to  the  objective-questions  and  for  the  status  of  the  objective. 
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Figure  2.  Schematic  of  FIRE 
results  reporting  forms. 
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There  is  a  set  of  the  first  four  forms  for  each  experiment  Thread.  The  fifth  form  is  for  Objective 
results,  which  is  a  fusion  of  the  results  from  all  of  its  Objective-Questions. 

A  Thread  is  an  Objective-Question  and  its  set  of  specific  measures, 
surveys,  situations,  and  data  needed  to  answer  that  Objective-Question. 

The  Thread  results  form  includes  measures,  processed  survey  answers,  the  complete  answer  to 
the  Objective-Question,  and  the  context  under  which  these  results  were  produced. 

The  Objective  results  fonn  is  different  than  the  other  forms  because  it  does  not  refer  to  a  Thread 
and,  hence,  does  not  contain  a  Thread  #  (see  below  for  a  description  of  the  Thread  #).  Also,  it  is 
in  the  FIRE  collaboration  workspace,  not  under  the  FIRE  Focus  Area  tab  as  are  the  other  forms. 

There  is  a  substantial  amount  of  infonnation  that  is  auto-filled  from  one  form  to  another  for  those 
forms  that  contain  a  Thread  #.  This  is  indicated  in  the  above  schematics.  The  information  that  is 
shared  is  shown  in  later  sections. 


1.1  Holistic  and  Sequential  Planning 

The  three  planning  fonns  are  set  up  for  both  sequential  and  holistic  planning. 

Sequential  Planning: 

Define  Objective,  input  to  FIRE  > 

Specify  Objective-Questions,  input  to  FIRE  > 

Etc.  down  to  data  to  be  captured. 

Later,  find  out  data  can’t  be  obtained  > 

Go  back  and  redo  > 

Iterate  until  done. 


Holistic  Planning: 

Look  across  the  full  spectrum  of  planning  elements  to  insure,  as  much  as  possible,  that 
the  full  Thread  can  be  accomplished. 

Insure  all  planning  elements  fit  to  produce  the  information  desired. 

Proceed  to  provide  input  to  FIRE. 

Later  modification  may  be  needed  as  things  change. 

Holistic  planning  usually  produces  an  adequate  plan  more  efficiently.  The  forms  are  structured 
and  linked  in  such  a  way  that  holistic  planning  is  facilitated.  The  basic  Objective  planning 
components  are  included  in  all  planning  forms,  can  be  edited  in  any  form,  so  that  one  does  not 
have  to  cycle  through  the  forms  to  insure  a  well  structured  and  achievable  Thread. 

Holistic  planning  can  be  done  at  any  point  in  the  planning  process,  and  applying  it  several  times 
is  beneficial.  At  the  outset  of  planning  the  following  aspects  should  be  considered: 

•  What  do  we  want  to  learn,  or  want  to  accomplish? 

•  Why  do  we  need  to  learn  that? 
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•  What  sort  of  information  do  we  need? 

•  What  data  do  we  need  to  capture  to  produce  that  information? 

•  What  sort  of  situation  is  needed,  and  what  events  need  to  occur,  to  produce  that  data? 
These  questions  are  linked.  Consider  all  of  these  aspects,  play  them  against  each  other  to  insure 
you  have  a  complete  overview  before  getting  into  sequential  planning  details.  Without  such  an 
overview,  much  planning  time  will  be  wasted  and  the  experiment  may  never  come  together 
correctly.  Take  opportunities  during  all  planning  phases  to  look  across  all  components  to  insure 
they  fit,  that  required  infonnation  can  be  produced. 
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Section  2.  Planning  Elements 

Planning  elements  are  those  quantities  that  define  the  information  needed  from  the  experiment, 
what  is  to  be  done  with  it,  and  the  required  results.  The  planning  elements  are: 

•  Objective 

•  Objective-Question(s) 

•  Attributes  and  Measures 

•  Survey  Questions 

•  Situations 

•  Events 

•  Data 

•  Analysis  Methods 

Several  results  are  reported.  They  are: 

•  Measures 

•  Processed  survey  question  answers 

•  Objective-Question  answer 

•  Context  and  its  impact  on  results 

•  Objective  status 

•  MUA  recommendations 

This  section  describes  each  of  these  elements  in  some  detail.  The  next  section  provides 
descriptions  of  the  input  format  and  content  for  each  to  be  provided  in  FIRE. 


2.1  Objective  and  Objective-Question 

Objectives: 

•  An  Objective  is  a  high-level  purpose,  usually  to  provide  an  operational  capability  or 
develop  a  system-level  capability. 

•  The  Objective  statement  does  not  address  specific  solutions  to  provide  that  capability. 
That  is  a  lower-level  consideration. 

•  A  Focus  Area  can  have  one  or  several  Objectives. 

Obj  ective-Questions : 

•  Objective-Questions  address  specific  elements  that  are  needed  to  achieve  the  Objective, 
to  provide  the  capability. 

•  An  Objective  can  have  one  or  several  Objective-Questions. 

•  Each  Objective-Question  addresses  specific  attributes  of  objective  development  that  are 
being  examined. 

•  The  attribute(s)  for  which  measures  are  to  be  obtained  must  be  contained  in  the 
Objective-Question  statement. 

•  Objective-Questions  often  address  specific  systems  or  processes. 

•  An  Objective-Question  asks  whether  a  needed  development  has  been  achieved. 
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•  An  Objective-Question  can  pose  only  one  question;  multiple-statement  questions  are  not 
allowed,  but  the  question  can  address  several  attributes/measures  (see  below). 

Objective-Questions  are  stated  in  such  a  way 
that  they  lead  directly  to  the  desired  measures. 


2.2  Experiment  Threads 

Each  Objective-Question  defines  an  experiment  Thread.  Each  Thread  consists  of  a  connected  set 
of  planning  components,  shown  in  Figure  3. 


\ 


.Experiment 

^Thread 


/ 


Figure  3.  Components  of  an  Experiment  Thread. 


Objective  results  report  the  status  of  the  Objective.  They  are  a  fusion  of  the  results  for  its 
component  Objective-Questions  (the  fusion  of  the  results  from  one  or  more  experiment  Threads). 
This  is  illustrated  in  Figure  4. 
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Figure  4.  Fusion  of  results  from  multiple  Threads  to  produce  Objective  result. 

2.2.1  Thread  Numbering 
Each  Thread  is  given  a  code:  E.g., 


Figure  5.  Thread  #  code. 

The  Thread  #  is  the  first  information  on  all  forms  and  summaries.  It  is  the  key  used  for  archiving 
and  retrieval  of  information  to/from  the  FIRE  database. 

Objective  Statement  Rule:  Threads  in  a  Focus  Area  that  have 
the  same  Objective  #  must  have  the  same  Objective  statement. 


2.3  MOP,  MOE,  Military  Utility,  and  Attributes 

Measures  are  experimentation’s  blood.  This  brief  discussion  sets  the  stage  for  how  measures  are 
handled  in  the  FIRE  forms.  The  terms  MOFE,  MOE,  and  MOP  are  in  common  use  and  we 
introduce  the  additional  measure:  MOU.  These  are 

•  MOFE  =  Measure  of  Force  Effectiveness 

•  MOU  =  Measure  of  Utility 

•  MOE  =  Measure  of  Effectiveness 

•  MOP  =  Measure  of  Performance 
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MOFE  is  used  when  determining  the  effectiveness  of  a  force’s  conduct  of  operations. 

MOU  is  a  measure  of  the  Utility  of  a  system  or  process  for  supporting  operational  activities. 

Whether  a  measure  is  an  MOE  or  MOP  is  not  fixed.  It  depends  on  the  specific  learning  that  is 
being  undertaken.  The  following  measures  descriptions  reflect  this  fact  in  the  two  different 
treatments  of  MOE.  This  use  is  what  is  set  up  in  FIRE. 

•  MOE  -  “Effective”  is  used  as  an  attribute  only  when  overall  effectiveness  of  system, 

people,  or  process  to  perform  its  stated  mission  is  to  be  evaluated. 

•  MOE  -  (other  than  Effective)  specific  measures  of  a  component  of  performance 

effectiveness.  This  may  be  a  roll-up  of  performance  measures. 

•  MOP  -  direct  measure  of  a  specific  performance  parameter  (people,  process,  or 
system). 


2.3.1  Attributes,  Measures,  and  Their  Relationship 

Attributes  and  measures  are  intimately  linked;  they  are  different  ways  of  expressing  the  same 
thing. 

•  Attributes  are  single-word  expressions  of  the  characteristics  of 
people,  things,  or  processes. 

•  Measures  provide  attribute  quantification. 

In  what  follows,  the  terms  attribute  and  measure  will  be  used  almost  interchangeably. 


Table  1  shows  the  three-level  attribute/measure  structure  that  is  in  use  for  experimentation  and 
implemented  in  FIRE. 


|~X~|Effective 


[xlMilitary  Utility 


Improved 


|~X~|l\leeded 


x 

Accessible 

X 

Reliable 

X 

Capable 

X 

Capacity 

X 

Robust 

X 

Sufficient 

X 

Available 

X 

Persistent 

X 

Flexible 

X 

Compatible 

X 

Secure 

X 

Accurate 

X 

Extensive 

X 

Assured 

X 

Timely 

X 

Efficient 

Usable 
Clear 
Trusted 
Manageable 
Relevant 
Compliant 


[^Applicable  |~X~|Desired 


Table  1.  Effectiveness  and  Military  Utility  attributes. 


The  FIRE  Objective  input  form  has  a  check  box  for  each  attribute,  used  to  indicate  which 
attributes  are  desired. 
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Following  the  measures  descriptions  shown  above,  attributes  are  used  as  follows: 


•  MOE  -  Effective  is  checked  when  a  roll-up  for  overall  effectiveness  is  desired. 


MOE 

-  These  measures 

are  the  components  of  effectiveness: 

o 

Accessible 

You  can  get  to  it. 

o 

Reliable 

It  is  there  when  needed. 

o 

Capable 

It/he/she/they  can  do  its  job. 

o 

Usable 

You  can  use  it. 

•  MOP  -  The  performance  measure  attributes  that  are  components  of  the  four  MOE  are 
listed  under  each  of  the  MOE  in  Table  1.  E.g.,  Robust,  Persistent,  Secure,  and  Assured 
are  components  of  Reliable. 

2.3.2  Military  Utility 

Military  Utility  is  a  special  attribute.  It  is  a  judgment  that  there  is  utility  (which  needs  to  be 
described)  in  military  use  of  the  system  or  process. 


•  MOLT  -  are  those  characteristics  that  are  used  to  determine  the  overall  utility  for 
supporting  operational  activities.  The  four  MOU  are: 

o  Improved  improves  the  perfonnance  of  operational  activities 

o  Needed  fills  a  gap  in  current  capabilities 

o  Applicable  can  be  applied  to  activity  perfonnance 
o  Desired  operational  personnel  want  the  capability 

•  Effectiveness  is  an  internal  attribute.  It  has  to  do  with  how  well  something  performs  its 
function. 


•  Utility  is  an  external  attribute.  It  has  to  do  with  how  well  something  contributes  to 
another  function,  in  this  case  to  a  military  activity. 


There  is  currently  no  MOP  equivalent  level  for  Military  Utility.  It  is  expected  that  most  of  the 
utility  detenninations  will  be  subjective. 


Objective  determinations  can  be  made,  e.g.,  the  number  of  times  a  capability  is  used  as  a 
measure  for  Desired.  MOP  under  the  MOU  attributes  will  be  developed  as  experience  allows. 


Two  rules  when  using  military  utility  as  an  attribute  are: 

•  The  specific  operational  activity  (task)  must  be  specified  in  the  Objective. 

•  It  must  be  a  separate  Thread.  It  is  not  appropriate  to  use  the  same  Thread  to  detennine 
effectiveness  and  utility. 
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3.  Measures,  Questions,  and  Data  Sources 

Each  attribute  contained  in  the  Objective-Question  has  an  associated  measures  and  data  tail. 

(The  tenn  “measures”  is  use  loosely  because  some  of  the  infonnation  to  be  gathered  will  be 
subjective  opinions.)  Where  the  data/infonnation  will  come  from  that  produces  each  measure 
has  to  be  defined. 

3.1  Quantitative  Measures  and  Subjective  Questions 

An  Attribute  specifies  what  is  to  be  detennined  but  not  specifics.  The  specific  detennination  can 
be  objective  or  subjective  or  both.  Data  sources  are: 

•  Objective  determinations  (quantitative  measures) 

o  System  Log 
o  Chat  Log 

o  Observer  logging  time-stamped  events 

•  Subjective  detenninations  (opinions) 

o  SME-observer  answering  attribute  questions 
o  Operator  answering  attribute  questions 

It  is  also  possible  to  process  subjective  determinations,  such  as  many  people  providing  answers 
to  survey  questions,  into  quantitative  measures  (such  as  averaged  responses  to  an  opinion  scale). 

For  any  Thread,  one  or  more  of  these  detenninations  can  be  specified.  Both  objective  and 
subjective  determinations  can  be  specified  for  the  same  attribute. 


3.2  Specifying  Measures 

Sufficient  information  about  the  measures  must  be  provided  so  that  data  to  be  captured  is  clearly 
specified.  This  means  specifying  exactly  what  is  to  be  measured  and  with  what  instrument.  E.g., 

Attribute  =  Timely  MOP  =  Timeliness 

Specific  measures:  a.  Time  from  submission  of  RFI  to  receipt  of  information. 

b.  Time  information  waits  in  queue  for  transmission. 

c.  RFI  processing  time. 

Each  of  these  measures  can  be  determined  by  data  from  system  or  event  logs. 
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Subjective  determinations  also  require  specificity,  which  is  done  by  providing  the  question  to  be 
asked.  E.g., 


Attribute  =  Timely  Subjective  Detennination 

Specific  questions:  a.  Was  the  information  timely? 

(poor  question,  no  specifics  provided  as  to  what  timely  means) 

b.  Was  mensuration  information  received  in  time  to  meet 
MAAP  cut-off? 

(good  question,  meaning  of  timely  is  well  defined) 

It  is  important  that  there  be  no  ambiguity  as  to  what  is  desired,  what  data  and  determination  are 
needed. 


3.3  Chat  as  a  Data  Source 

Chat  has  multiple  uses  in  experimentation. 

•  Real-time  experiment  control. 

•  Real-time  operator  collaboration. 

•  Information  source  for  experiment  analysis. 

•  Data  source  for  experiment  analysis. 

•  Experimented  with  for  its  own  capabilities  determination  (same  as  any  other  system). 

•  Source  for  context  information. 

The  use  of  Chat  for  data  and  information  acquisition  is  complex  because  very  time-consuming 
mining  and  extraction  are  required.  Context  infonnation  is  somewhat  simpler  to  extract  but  is 
still  time-consuming.  The  major  key  to  extraction  is  correlating  time  stamps  with  data  and 
information  events  of  interest. 

Because  of  the  work  involved  in  extracting  information  from  Chat  it  is  important  to  be  careful  to 
specify  exactly  what  is  to  be  extracted  and  design  the  experiment  event  so  that  the  needed 
information  in  the  Chat  log  can  be  easily  located.  Two  examples  follow. 

Example  1 :  Chat  is  specified  to  determine  the  time  lapse  between  collaboration  events. 
This  is  straightforward  and  reasonable.  One  does  have  to  insure  that  the  specified  events  are 
addressed  during  the  appropriate  Chat  session. 

Example  2:  An  appropriate  question  for  an  experiment  could  be  “Is  COP  information 
clear?”  One  could  look  in  Chat  to  attempt  to  answer  that  question.  No  further  specification  than 
the  question  makes  obtaining  the  information  difficult  and  open  to  interpretation.  It  is  best  to 
instruct  the  analyst  in  what  to  look  for  with  a  carefully  crafted  measure. 
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E.g.  Measure  =  number  of  instances  of  requests  for  clarification  of  information. 


The  analyst  has  to  go  into  the  Chat  log  and  look  for  those  instances,  but  a  measure  is  now 
specified  and  what  is  to  be  looked  for  is  specific. 

Context  Mining :  Chat  is  very  useful  for  this  purpose.  One  often  looks  for  instances  of  systems 
down,  operator  overload,  operator  confusion. 

Operational  Activity/Human  Performance :  Chat  contains  a  wealth  of  information  about 

•  How  well  operational  activities  are  proceeding. 

•  How  well  humans  are  performing  their  tasks. 

•  The  level  of  SA/SU. 

Infonnation  can  be  extracted  about  these  and  related  matters.  As  noted  above,  such  extraction  is 
difficult  and  well-crafted  measures  must  be  specified  or  the  information  extracted  will  be  suspect 
and  of  limited  value. 

Conclusion:  The  first  reaction  to  using  Chat  as  an  experiment  data  source 
should  be  to  not  do  so.  If  the  decision  is  made  to  do  so,  carefully  craft  the 
measures  to  be  detennined  and  how  the  appropriate  information/data  will 
be  extracted  as  a  prerequisite  to  using  this  source. 
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Section  4.  Analysis  Planning 

Experiment  Analysis  can  be  simple  or  extremely  complicated,  depending  on  the  number  of 
measures  to  be  evaluated  and  their  types.  This  section  shows  the  types  of  analyses  that  are 
required  for  various  measures  specifications. 

Important  to  remember  in  analysis  planning  is  that  everything  points  toward  answering  a  specific 
question.  One  is  dealing  with  an  inverted  tree,  as  shown  for  a  simple  case  in  Figure  6. 


Figure  6.  Analysis  process  to  distill  measured  results  into  Objective-Question  answer. 

If  a  combination  of  subjective  and  objective  measures  are  specified,  the  analysis  can  become 
quite  complicated.  Truly  frightening  is  Figure  7  at  the  end  of  this  section,  which  shows  how 
complex  analysis  can  become  if  planners  turn  themselves  loose  and  plans  for  multiple  subjective 
and  objective  measures  for  a  Thread. 


4.1  MOE  and  MOP  Analysis  Implications 

The  choice  of  attributes  and  measures  has  implications  for  analyses  to  be  done.  Some 
consequences  from  checking  various  attributes  are: 

•  If  a  single  MOP  is  checked,  only  its  measures  are  required,  but  there  may  be  more  than 
one  measure  (e.g.,  there  can  be  several  time  measures  for  timely). 

o  There  can  be  more  than  one  type  of  determination  for  the  MOP.  Both  an 
objective  measure  and  a  subjective  opinion  could  be  specified. 

•  If  multiple  MOPs  are  checked,  the  above  applies  for  each. 

•  If  an  MOE,  and  only  the  MOE,  is  checked  a  subjective  opinion  will  be  specified  because 
no  MOP  is  being  specified  to  support  it. 

•  If  an  MOE  is  checked,  and  also  some  of  its  MOPs,  then 

o  MOP  measures  are  to  be  determined  and  a  roll-up  of  those  measures  into  the 
MOE  determination  is  to  be  done. 
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o  It  is  also  possible,  in  addition,  to  specify  a  subjective  determination  of  the  MOE. 

•  If  Effective,  and  only  Effective,  is  checked  a  subjective  opinion  will  be  specified  because 
no  supporting  MOEs  or  MOPs  are  to  be  determined. 

•  If  more  attributes  are  checked  with  Effective  then 

o  At  least  one  of  them  must  be  an  MOE. 
o  In  this  case  a  full  roll-up  to  effectiveness  is  required, 
o  The  rules  above  apply. 


4.2  Analysis  Planning  Components 

Analysis  is  the  most  difficult  part  of  an  experiment  to  plann  because  so  many  details  are 
required.  Specific  analyses  must  be  planned  for: 

•  Each  measure,  and  fusion  if  needed 

•  Each  survey  question,  and  fusion  if  needed 

•  Each  Objective-Question 

•  The  Objective 

•  Context 


The  most  difficult  analysis  to  conduct  is  a  roll-up  of  different  types  of  information.  Consider  a 
particular  attribute  that  has  several  measures  and  questions  associated  with  it.  The  following 
could  be  needed: 


Attribute  1 

Measure  la 
Measure  lb 
Question  1A 
Question  IB 


Two  measures  from  different  sources. 
Need  procedure  for  combining. 


Measure  Value 
Measure  Value 
Question  Answer 

Question  Answer  Same  deal  for  combining  question  answers. 
Attribute  Roll-Up  of  all  the  above 


And  there  may  be  several  attributes,  requiring  their  roll  up  for  the  Objective-Question.  Thus  all 
of  the  following  could  be  required: 

•  Attribute:  roll  together  measures  and  answers  into  single  determination. 

•  Objective-Question:  roll  together  attributes  into  single  answer/detennination. 

•  Objective:  roll  together  Objective-Question  results  into  Objective  status. 


If  all  of  the  above  are  required  for  an  Objective  with  many  Threads,  and  complex  Objective- 
Questions  the  analysis  can  be  quite  complex.  Avoiding  overly  complex  analyses  is  a  good 
reason  to  keep  planning  elements  simple. 


The  following  figure  diagrams  the  analysis  procedures.  Full  complexity  is  illustrated. 


Only  two  threads  are  shown  for  the  objective,  there  can  be  any  number. 
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All  possible  types  of  data  that  produce  information  for  Thread- 1  are  shown.  There  are  almost  no 
cases  where  all  types  are  needed. 

The  complex  analysis  process  needed  to  fuse  many  types  of  information 
is  an  argument  for  keeping  the  number  of  types  small.  Keep  it  simple. 


Figure  7.  Analysis  plan  structure,  overly  complex  case. 
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Section  5.  Results  Elements 


There  are  three  phases  of  results  reporting: 

•  Production  of  measures,  processed  question  answers,  and  context. 

•  Analyses  to  produce  Objective-Question  answers,  Objective  status,  and  context  impact. 

•  Report  development:  quick  look,  program  reports,  experiment  final  report. 

Results  production  is  a  product  of  analyses,  so  the  information  in  this  section  closely  follows  that 
in  the  former.  It  is  duplicated  included  here,  more  briefly,  so  that  this  section  stands  on  its  own. 

The  first-level  results  to  be  produced  are: 

•  Each  measure  specified 

o  System  log  measures 
o  Chat  log  measures 
o  Observer  event  measures 

o  If  more  than  one  measure  source  is  specified,  a  fusion  of  those  measures  may  be 
needed. 

•  Each  question  asked,  with  specific  reference  to  the  attribute 

o  SME  observer  answers 
o  Operator  answers 

o  If  more  than  one  answer  source  is  specified,  a  fusion  of  those  answers  may  be 
needed. 

o  Answers  may  be  processed,  such  as  average  response  on  a  scale. 

•  Context/situation  under  which  information/data  was  obtained 

o  System  status 
o  Operator  status 
o  Procedures  used 

The  second-level  results  to  be  produced  are: 

•  Objective-Question  answer,  with  elaboration 

•  Context  impact  on  the  Objective-Question  answer. 

The  third-level  results  to  be  produced  are: 

•  Objective  status 

•  Context  impact  on  Objective  status 

•  Recommendation  to  the  MUA  board. 

All  of  the  above  results  are  input  to  FIRE.  The  first  two  levels  are  placed  in  FIRE  forms.  The 
third  level  is  in  spreadsheets  in  the  TACFIRE  TW  workspace. 
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4.1  Planning  to  Reporting  Process 


The  following  figure  shows  how  various  planning  components  can  be  used  to  produce  the  final 
report.  Thorough  writing  during  planning  can  greatly  aid  final  report  production. 


Pre  Expt.  Writing 


Planning  to  Reporting  Structure 


Experiment 

Purpose 


Experiment 

Venue 


Objectives 

Rationale 


Object.  Quest. 
Attrib.  Meas. 


Situations 


During  Expt.  Reporting 


MEL 


Deviations 
from  the  MEL 


Daily  System 
Status  Report 


T 

Context 

Development 


I 


Data  Archiving 


Analysis 

Work 


Data-Thread  Links 


All  of  the  Analysis 
Work  activities  (in 
pink)  have  to  be 
planned  in  the 
Analysis  Plan. 


Figure  8.  Reporting 


Objective  Result 
&  Context  Rollup 


Reporting 


Actual 

Experiment 

Conduct 
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Section  6.  Workflow:  FIRE  and  TACFIRE  Workspace 

There  are  two  principal  areas  where  experiment  planning  and  reporting  work  occur. 

•  FIRE  -  Standard  planning  forms,  entry  of  prescribed  planning  and  reporting  elements. 

•  TACFIRE  -  Workspace  containing  folders  for  communication,  living  spreadsheets, 

documents,  minutes. 

The  FIRE  forms  have  been  described  generally  above.  The  workspace  has  a  set  of  prescribed 
folders  that  are  set  up  by  the  FIRE  managers  for  specific  uses.  Users  who  have  the  appropriate 
authorization  may  create  other  folders  within  this  prescribed  structure,  as  needed.  It  is  important 
that  the  prescribed  folders  be  used  for  their  specified  purposes,  that  the  assigned  infonnation  not 
be  placed  in  other  folders. 

Figure  9  illustrates  how  fonns  and  folders  are  used  throughout  the  experimentation  process.  For 
Forms  labeled  “By  Focus  Area”  and 
Folders  labeled  “Focus  Area  Folders” 
there  is  a  form  set  or  included  folder  for  each  Focus  Area. 

Folder  content  descriptions: 

Meetings:  Meeting  invitations,  descriptions,  minutes,  presentations. 

Focus  Area:  One  folder  for  each  Focus  Area  for  archived  communications,  documents, 
files,  things  pertinent  specifically  to  that  Focus  Area.  Sub-folders  may  be  created  for  a  Focus 
Area  if  needed. 

Plans:  Specific  planning  documents  that  need  to  be  shared  and  updated  as  the  experiment 
is  being  developed.  Most  of  the  documents  will  be  spreadsheets.  Included  are  the  Install  Matrix, 
Manning  Matrix,  Master  Event  List,  etc. 

Survey  Builder:  Surveys  are  currently  built  in  Survey  Builder.  Whether  this  folder  will 
be  needed  is  uncertain. 

Data:  All  Data  is  placed  in  a  folder  for  the  Focus  Area  for  which  it  was  generated.  There 
are  sub-folders  for  each  data  type:  system,  chat,  observer,  survey.  There  is  also  a  folder  for  daily 
summaries  (any  type  such  as  daily  equipment  status,  SME  observations,  MEL  deviations). 

Results:  Each  Focus  Area  has  a  folder.  These  are  Objective-level  results.  The  folders 
contain  spreadsheets  into  which  are  entered  Objective  status,  context  impact,  and  MUA 
recommendations. 

Experiment  Documents:  General  documents  that  describe  the  purpose,  overall  structure, 
guidelines  for  the  experiment  as  a  whole. 


21 


A  separate  document  provides  directions  for  access  to  and  use  of  the  workspace. 
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Section  IV.  Access  to  FIRE  Forms  and  Summaries 


Each  of  the  forms  shown  schematically  in  Figure  1  are  in  FIRE  (except  for  Objective  results 
which  are  in  the  TACFIRE  workspace).  For  each  there  is  an  input/edit  form  and  a  summary 
report  that  shows  the  information  currently  in  the  database.  This  section  describes  these  forms 
and  summaries  and  how  to  access  them.  The  following  section  provides  directions  for  the 
various  inputs. 

There  are  two  user  types  and  they  have  different  access.  Referring  to  Figure  10,  access  is  as 


follows: 

•  Create  New  Thread  Lead  only 

•  Print  Friendly-Lead  View  Lead  only 

•  Print  Friendly-General  View  All 

•  Form  Edit  Link  (on  summary  form)  Lead  only 

•  Analysis  Planning  Tab  Lead  only 


If  a  general  user  attempts  to  access  a  area  for  which  they  are  not  authorized,  a  prompt  will  appear 
that  says  access  denied. 

Figures  10  and  1 1  show  screen  captures  of  the  Objective  planning  summary  form  and  input/edit 
form.  Text  is  included  that  shows  input  to  be  provided  for  this  particular  set.  The  Thread  #  used 
for  this  illustration  is  C2-01.01. 

This  fonn  is  accessed  as  follows: 

•  Click  on  the  “TW08”  link  in  the  left,  blue,  navigation  bar. 

•  Click  on  the  “Focus  Areas”  tab  at  the  top  of  the  page. 

•  Click  on  the  next-level  red  tab  for  the  Focus  Area  desired. 

•  The  Objective  Summaries  page  for  that  Focus  Area  will  appears. 

Navigation  to  the  various  forms  and  summaries  occur  from  this  level. 

•  To  start  a  new  Thread:  click  on  the  “Create  New  Thread”  link. 

•  To  access  ThreadEx  or  Results  forms  and  summaries:  click  on  the  appropriate  blue  tab. 

•  To  access  the  edit  form  for  a  Thread:  click  on  the  “Form  Edit  Link”  number  in  the 
summary. 

Reviews  appear  only  at  the  end  of  the  edit  form  and  in  the  Printer  Friendly-Lead  View.  The 
general  viewer  summaries  do  not  contain  these  reviews. 

Index  Number  -  Whenever  a  new  Thread  is  created  it  is  assigned  an  Index  #.  This  number 
appears  at  the  top  of  all  summaries  and  is  the  “Fonn  Edit  Link”. 


23 


Figure  10.  Screen  capture  of  a  FIRE  summary  form. 


The  edit  form  for  this  Thread  is  shown  in  Figure  1 1 .  The  full  form  is  not  shown. 

The  form  is  divided  into  logical  sections  with  blue  headers  above  each  section.  E.g.,  quantitative 
measures  are  a  section.  Some  of  the  headers  have  brief  directions  for  the  input  to  be  provided. 

The  proper  format  for  the  Thread  #  is  shown.  If  this  format  is  not  followed  the  number  will  be 
rejected.  Duplicate  numbers  will  also  be  rejected. 

Check  boxes  for  the  desired  Attributes  are  in  the  middle  of  the  form.  Accessible,  Accurate,  and 
Timely  have  been  checked.  This  indicates  that  a  subjective  question  about  accessibility  and 
measures  for  accurate  and  timely  are  desired. 

The  attribute  check  boxes  appear  only  in  the  objective  input/edit  form.  Those  attributes  to  be 
examined  are  typed  into  the  text  areas,  as  shown,  and  this  is  where  they  are  seen  in  subsequent 
forms  and  reports. 
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Figure  11.  Initial  Thread  input  form  and  Objective  edit  form. 
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7.1  Status  Report 


The  status  of  planning  and  of  results  development,  for  each  Focus  Area,  is  shown  under  its  Status 
tab.  Shown  are  the  status  chosen  from  the  drop-down  in  the  planning  input/edit  form.  Table  2  is 
a  mock-up  of  the  summary. 


Status 

(Focus  Area  Name) 

Report 

Thread 

Objective 

ThreadEx 

Analysis 

Results 

Complete 

Complete 

Modified 

Reviewed 

Reviewed 

Etc. 

Table  2.  Schematic  of  FIRE  status  report. 
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8.  FIRE  Forms  and  Summaries  Content 


Hybrid  mock-up  displays  of  the  input/edit  forms  and  summaries  are  presented  in  Tables  3-6. 
Real  FIRE  forms  are  shown  only  in  Figures  10  and  11.  The  following  format  is  used: 

•  Text  area  titles  are  in  black  (or  blue  if  linked  to  other  forms). 

•  Headers  are  in  red.  (Headers  are  only  in  the  input/edit  forms.) 

•  Status  drop-down  and  reviewer  comment  text  areas  are  the  same  in  all  forms,  thus  are 
shown  only  in  the  Objective  Planning  display. 

Those  text  areas  that  are  linked  to  those  in  other  forms  are  so  indicated.  All  of  the  information  in 
the  Objective  summary  is  replicated  to  the  other  forms  except  status. 

Those  text  areas  that  are  linked  between  forms  can  be  edited  in  any  form  and  the  edited  text  will 
be  updated  in  all  forms  and  summaries.  This  facilitates  holistic  planning. 


8,1  Objective  Planning  FIRE  Form  and  Summary 

Status  is  input  by  three  choices  from  a  drop-down.  “Reviewed”  means  that  the  reviewer  wants 
modifications  made.  “Modified”  means  that  the  lead  has  made  the  recommended  modifications. 
“Complete”  means  that  the  reviewer  is  satisfied  with  the  input.  The  reviewer’s  comments  are 
provided  at  the  end  of  the  form 

The  other  information  to  be  input  has  been  well  described  earlier  in  this  report.  Very  brief 
information  is  provided  below. 

There  is  a  text  area,  with  directions  for  the  Objective  and  Objective-Question. 

The  next  section  contains  the  check-boxes  for  attributes. 

This  is  followed  by  a  section  for  the  three  types  of  quantitative  measures: 

•  System  Log 

•  Chat  Log 

•  Observer  Log 

The  last  section  before  the  reviewer’s  comments  is  for  subjective  opinions: 

•  SME  Observer  questions 

•  Operator  survey  questions 
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Objective  Planning  Spreadsheet 

Thread  # 

Linked  to  other  forms 

Status 

Indicate  the  current  status  by  choosing  correct  drop-down. 

Reviewed  Modified  Complete 

Objective:  1  sentence,  Operational  or  Development,  same  #  =  same  statement 

Objective  |Linked  to  other  forms 

Qbjective-Question:  1  sentence,  single  question,  include  attributes. 

Objective-Question  Linked  to  other  forms 

Effective  Attributes:  Attributes  for  the  Effectiveness  Measures  to  be  determined.  Indicate  by  checking. 

Check  "Effective"  when  effectiveness  rollup  is  desired,  etc.  for  its  top  4  attributes. 

Effective 

Accessible  Reliable  Capable  Usable 

Capacity  Robust  Sufficient  Clear 

Available  Persistent  Flexible  Trusted 

Compatible  Secure  Accurate  Manageable 

Extensive  Assured  Timely  Relevant 

Efficient  Compliant 

Military  Utility  Attributes:  Attributes  for  the  Effectiveness  Measures  to  be  determined.  Indicate  by  checking. 

Check  "Military  Utility"  when  utility  rollup  is  desired. 

Military  Utility 

Improved  Needed  Applicable  Desired 

Specific  QUAN' 
System  Log  Measures 

Chat  Log  Measures 

Observer  Log 

Measures 

TITATIVE  Measures  with  their  Data  Source 

Linked  to  other  forms 

Linked  to  other  forms 

Linked  to  other  forms 

Specific  SUBJECT] 
SME  Observer 
Questions 

Survey  Questions 

VE  /  QUALITATIVE  Questions  with  their  Information  Source 

Linked  to  other  forms 

Linked  to  other  forms 

Reviewer's  Comments. 

Objective  and 
Objective-Question 
Review  Comments 

Choose  Modified  status  when  review  changes  have  been  made. 

Attributes,  Measures, 
Questons  Review 
Comments 

Table  3.  Objective  Planning  hybrid  representation. 
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8.2  ThreadEx  FIRE  Form  and  Summary 

Planning  for  the  events,  conditions,  and  data  to  be  captured  is  done  using  the  ThreadEx  input/edit 
form  and  summary.  (ThreadEx  means  definition  of  the  details  of  the  Thread  for  the  experiment.) 
This  planning  is  to  set  up  the  proper  circumstances  so  that  the  data  needed  to  evaluate  the 
measures  can  be  obtained.  Figure  8  is  the  mock-up  of  the  FIRE  ThreadEx  format  and  following 
are  general  explanations  of  its  various  planning  sections. 

The  Objective,  Objective-Question,  and  the  desired  measures  and  questions  are  repeated  in  this 
form  for  reference  during  this  stage  of  planning  (and  so  that  modifications  to  them  can  be  made 
in  this  form  rather  than  having  to  return  to  the  Objective  fonn). 

The  first  input  section  provides  description  of  the  events. 

•  General  Description 

•  Date 

•  Thread  Frequency/Duration 

This  infonnation  is  used  in  forming  the  experiment  Master  Event  List  (MEL). 

Next  are  the  conditions. 

•  Operational  Conditions  -  what  the  operating  forces  are  to  be  doing  and  where. 

•  System  Conditions  -  what  systems  are  to  be  in  use  and  how  configured. 

•  Information  Conditions  -  what  infonnation  is  to  be  “flowing”  and  its  characteristics. 

Finally  are  participant’s  instructions.  These  are  directions  for  what  they  are  to  do  during 
experiment  execution,  and  also  immediately  after  for  that  part  of  reporting  that  immediately 
follows  experiment  completion. 

•  Participant  Status  -  where  to  be,  when. 

•  Instructions/Procedures  -  what  to  do. 

•  Data  Collection  Instructions  -  what  media  for  collection,  how  to  transport,  where  to  store. 

•  Reporting  Instructions  -  daily  status,  next-day  needs,  post-experiment  infonnation 
delivery. 
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ThreadEx 

Thread  # 

Linked  to  other  forms 

Objective 

Linked  to  other  forms 

Objective-Question 

Linked  to  other  forms 

ThreadEx  Coordinator 

General  Description 

Date 

Thread  Frequency/Duration 

Conditions  to  be  s 

Operational  Conditions 

et  so  needed  data  can  be  captured. 

System  Conditions 

Information  Conditions 

Data  /  information  to 

System  Log  Measures 

be  captured  for  specific  measures/questions,  by  type. 

Linked  to  other  forms 

Chat  Log  Measures 

Linked  to  other  forms 

Observer  Log  Measures 

Linked  to  other  forms 

SME  Observer  Questions 

Linked  to  other  forms 

Survey  Questions 

Linked  to  other  forms 

Instructions  to  e. 

Participant  Status 

xperiment  participants. 

Instructions  /  Procedures 

Data  Collection  Instructions 

Reporting  Instructions 

Table  4.  ThreadEx  hybrid  representation. 
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8.3  Analysis  Planning  FIRE  Form  and  Summary 


Analysis  specification  can  be  lengthy  because  of  the  need  to  produce  several  types  of  results. 
Lowest-level  results: 

•  Measures  values. 

•  Question  answers. 

•  Fusion  of  measures  and  answers  into  attribute  evaluations. 

•  Determination  of  the  context/situation/special  circumstances  that  were  encountered. 
Higher-level  results: 

•  Objective-Question  answer  through  roll-up  of  attribute  evaluations. 

•  Impact  of  context  on  the  Objective-Question  answer. 

•  Objective  status. 

•  Impact  of  context  on  the  Objective  status. 

For  each  of  the  measures  and  questions  the  procedures  for  their  determination,  procedure  status, 
and  the  person  responsible  for  carrying  out  the  analysis  are  provided.  The  same  planning  is 
provided  for  the  Objective-Question  and  the  Objective. 

CONTEXT: 

An  important  part  of  analysis  is  to  determine  the  context/conditions  under  which  the  information 
was  obtained: 

•  Conditions  under  which  information  was  captured  for  each  Thread. 

•  Deviations  from  the  planned  conditions  are  the  most  important  to  record. 

•  Impact  of  these  conditions  on  the  results  for  each  measure,  if  any. 

•  These  conditions  provide  the  overall  context  for  the  Thread. 

•  Context  impact  on  the  Objective-Question  (validity,  caveats,  etc.). 

•  Context  impact  on  the  Objective  (roll-up  of  impacts  on  the  individual  Objective- 
Questions). 

ANALYSIS  STATUS: 

Analysis  status  is  slightly  different  than  the  status  for  Objective  Planning  and  ThreadEx.  Status 
for  Objective  and  ThreadEx  is  for  quality  of  the  planning.  For  Analysis,  status  is  provided  for 
both  the  quality  of  the  analysis  plan  and  for  status  of  results  production.  Reviewer  comments 
switch  from  one  to  the  other  once  the  experiment  is  completed  and  analysis  begun. 
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Analysis  Planning 

Thread  #  Linked  to  other  forms 

Procedures  for  producing 
Measures  from  system  logs. 
System  Log  Measures 

3  specific  quantitative  measures. 

Linked  to  other  forms 

System  Analysis  Procedures 

System  Analysis  Status 

System  Analyst 

Measures  from  Chat  logs. 

Chat  Log  Measures 

Linked  to  other  forms 

Chat  Analysis  Procedures 

Chat  Analysis  Status 

Chat  Analyst 

Measures  from  Observer  logs. 
Observer  Log  Measures 

Linked  to  other  forms 

Observer  Measures  Analysis  Procedures 

Observer  Measures  Analysis  Status 

Observer  Measures  Analyst 

Procedures  for  producing  qualil 

Attribute  results  from  SMI 

SME  Observer  Questions 

tative  attribute  results  through  questions. 

E/Observer  answers  to  questions. 

Linked  to  other  forms 

SME  Questions  Analysis  Procedures 

SME  Questions  Analysis  Status 

SME  Questions  Analyst 

Attribute  results  from  op 
Survey  Questions 

srator  answers  to  survey  questions. 

Linked  to  other  forms 

Survey  Question  Analysis  Procedures 

Survey  Analysis  Status 

Survey  Analyst 

Procedures  for  aggregating  measures  a 

Aggregation  Procedures 

nd  question  results  into  the  Objective-Question  answer. 

Thread  Context  Procedures 

Objective-Question 

Linked  to  other  forms 

Objective-Question  Analysis  Procedure 

Objective-Question  Analysis  Status 

Objective-Question  Analyst 

Procedures  for  aggregating  Objectivt 
Objective 

2-Question  answers  into  Objective  result. 

Linked  to  other  forms 

Objective  Analysis  Procedure 

Objective  Analysis  Status 

Objective  Analyst 

Table  5.  Analysis  planning  hybrid  representation. 
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8.4  Results  Reporting  FIRE  Form  and  Summary 

Results  are  realized  through  the  planned  analyses,  so  their  components  closely  follow  analysis 
components.  Thus,  what  is  shown  immediately  below  for  results  fonn  input  has  the  same  titles 
as  for  analysis,  except  here  it  is  the  values  that  are  provided  vice  the  process  to  produce  them. 

Basic  results  are: 

•  Measures  values. 

•  Question  answers. 

•  Fusion  of  measures  and  answers  into  attribute  evaluations. 

•  Determination  of  the  context/situation/special  circumstances  that  were  encountered. 

The  next  levels  of  results  are: 

•  Objective-Question  answer  through  roll-up  of  attribute  evaluations. 

•  Impact  of  context  on  the  Objective-Question  answer. 

•  Objective  status. 

•  Impact  of  context  on  the  Objective  status. 


Results 

Thread  # 

Linked  to  other  forms 

List  of  all  quantitative 

System  Log  Measures 

System  Log  Measures  Values 

Chat  Log  Measures 

Chat  Log  Measures  Values 

Observer  Log  Measures 

Observer  Log  Measures  Values 

measures  that  were  produced  by  analysis. 

Linked  to  other  forms 

Linked  to  other  forms 

Linked  to  other  forms 

List  of  all  subjective  re 

SME  Observer  Questions 

SME  Observer  Answers 

Survey  Questions 

Survey  Answers 

suits,  by  question,  that  were  produced  by  analysis. 

Linked  to  other  forms 

Linked  to  other  forms 

Context  when  information  was  ca[ 

Thread  Context 

Objective-Question 

Objective-Question  Answer 

Context  Impact 

>tured,  Objective-Question  answer,  and  context  impact. 

Linked  to  other  forms 

Table  6.  Objective-Question  results  hybrid  representation. 
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Section  9.  Input  Field’s  Content  Directions 

Following  are  tables  that  contain  brief  directions  for  the  inputs  to  be  provided  in  each  text  field. 
Each  table  is  for  one  input/edit  fonn. 

•  When  formats  are  shown  they  must  be  followed. 

•  Those  inputs  in  blue  and  bold  are  shared  across  multiple  forms  and  may  be  edited  in  any 
of  the  forms  in  which  they  appear. 

Example  Threads  with  correct  planning  content  are  presented  in  Section  10. 
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9.1  Objective  Planning  Input  Directions 


Objective  Planning  Input  Directions 

Thread  # 

Thread  identifier  using  specified  format. 

Status 

Planning  components  status. 

Choose  status  from  the  drop-down. 

Only  Reviewer  can  choose  Reviewed  or  Complete. 

Reviewed  =  review  has  been  conducted  and  modifications  are  needed. 

Modified  (Lead  or  SME  use  only)  =  modifications  have  been  made  in  response  to 
reviewer’s  comments  and  another  review  is  needed. 

Short  Title 

Two  or  three  word  title,  used  as  a  shorthand  reference  to  this  Thread. 

Each  Thread  must  have  a  unique  Short  Title. 

Objective 

High-level  statement  of  an  operational  or  developmental  purpose. 

Only  a  single  sentence  allowed. 

All  Objectives  with  the  same  Objective  number  must  have  the  same  statement. 

Details  of  specifics  to  be  determined  go  in  the  Objective-Question. 

An  Objective  can  have  several  Objective-Questions. 

Objective- 

Question 

Each  Objective-Question  contributes  to  achieving  the  Objective. 

There  may  be  several  Objective  Questions  for  an  Objective. 

Only  a  single  question  is  allowed  for  each  Thread.  If  more  than  one  question  is  needed,  use 
multiple  threads. 

The  attributes  for  the  measures  to  be  obtained  must  be  in  the  question.  Sufficient  detail  is 
provide  in  the  question  to  guide  formulating  needed  measures. 

Attributes 

Check  the  attributes  for  which  measures  are  to  be  determined. 

This  information  is  for  reference  when  providing  measures  input  below. 

It  is  not  replicated  on  other  sheets. 

System  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  electronic  system  logs. 

Specify  the  system  log  from  which  data  will  be  obtained. 

Format  "Attribute:  System:  measure  description,  second  measure  description;", 
e.g.,  Timely:  GCCS:  elapsed  time  between  information  sent  and  received,  elapsed  time  between 
information  received  and  displayed. 

Chat  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  Chat  logs. 

Specify  the  Chat  Room  from  which  the  log  is  to  be  obtained. 

Format  "Attribute:  Chat  Room:  measure  description;  second  measure  description;  etc", 

e.g.,  Reach:  IRS  Analysts:  number  of  people  using  chat  as  a  function  of  situation. 

Efficiency:  IRS:  number  of  mouse  clicks  required  to  log  on. 

Observer  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  SME-Observer  logged  event 
information. 

Specify  the  logger  who  is  to  record  the  information. 

Format  "Attribute:  Logger:  measure  description,  second  measure  description;". 

e.g.,  Timely:  UAV  Observer:  elapsed  time  between  request  for  imagery  and  target  obtained, 
elapsed  time  between  target  obtained  and  image  chipped,  elapsed  time  from  image  obtained  and 
image  sent. 

SME  Observer 
Questions 

Specific  questions  to  be  answered  by  an  SME-Observer  (SUBJECTIVE). 

Specify  the  SME. 

Format  "Attribute:  SME:  question,  second  question;". 

e.g.,  Timely:  CIC  Observer:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  Observer:  Was  there  sufficient  information  to  provide  the  needed  situation 
assessment? 

36 


Survey 

Questions 

Specific  questions  to  be  answered  by  operator  survey  (SUBJECTIVE). 

Specify  the  operators  to  be  surveyed. 

In  most  cases  the  SME  and  Operator  questions  will  be  the  same. 

Format  "Attribute:  personnel:  question,  second  question;".  In  most  cases  the  SME  and 

Operator  questions  will  be  the  same. 

e.g.,  Timely:  CIC  personnel:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  analysts:  Was  there  sufficient  information  to  provide  the  needed  situation 
assessment? 

Obj  &  Obj-Quest  Reviewer  guidance  for  needed  input  modifications.  This  review  is  for  Short  Title  through 
Review  Objective-Question.  Choose  the  appropriate  status  drop-down  after  review  or  modifications. 


Comments 

Attrib,  Meas, 
Quest  Review 
Comments 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  Attributes,  Measures,  and  Questions. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Table  7.  Directions  for  Objective  planning  input. 
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9.2  ThreadEx  Input  Directions 


ThreadEx  Input  Directions 

Thread  # 

Thread  identifier  using  specified  format. 

Status 

Planning  components  status. 

Choose  status  from  the  drop-down. 

Only  Reviewer  can  choose  Reviewed  or  Complete. 

Reviewed  =  review  has  been  conducted  and  modifications  are  needed. 

Modified  (Lead  or  SME  use  only)  =  modifications  have  been  made  in  response  to 
reviewer’s  comments  and  another  review  is  needed. 

Short  Title 

Two  or  three  word  title,  used  as  a  shorthand  reference  to  this  Thread. 

Each  Thread  must  have  a  unique  Short  Title. 

Objective 

High-level  statement  of  an  operational  or  developmental  purpose. 

Only  a  single  sentence  allowed. 

All  Objectives  with  the  same  Objective  number  must  have  the  same  statement. 

Details  of  specifics  to  be  determined  go  in  the  Objective-Question. 

An  Objective  can  have  several  Objective-Questions. 

Objective-Question 

Each  Objective-Question  contributes  to  achieving  the  Objective. 

There  may  be  several  Objective  Questions  for  an  Objective. 

Only  a  single  question  is  allowed  for  each  Thread.  If  more  than  one  question  is  needed,  use 
multiple  threads. 

The  attributes  for  the  measures  to  be  obtained  must  be  in  the  question.  Sufficient  detail  is 
provide  in  the  question  to  guide  formulating  needed  measures. 

ThreadEx  Coordinator 

Person  responsible  for  conduct  of  this  Thread  during  the  experiment. 

It  may  or  may  not  be  the  Initiative  Lead. 

Format  Name;  Email  address  (SIPR  &  NIPR);  Land  line  number;  Mobile  number. 

General  Description 

General  description  of  the  events  to  occur  so  that  thread  data  thread  can  be  captured. 

1.  What  you  are  going  to  do?  2.  What  do  you  want  to  accomplish?  3.  What  do  you  expect  to  learn? 

Date 

Dates  during  which  you  want  data  captured  for  this  Thread. 

Experiment  ThreadExs  must  be  performed  on  a  specific  date(s)  and  time(s). 

No  free  play! 

The  times  can  either  be  Local  or  Zulu  as  directed  by  the  Experiment  Director. 

THIS  INFORMATION  WILL  BE  USED  TO  POPULATE  THE  MEL. 

Thread  Frequency/DurationTimes  during  the  day  for  data  capture 

or  the  frequency  during  the  day  of  data  capture. 


Operational  Conditions 

What  operating  forces  are  to  be  doing  so  that  the  correct  data  is  developed  and 
captured. 

Operational  Conditions  reflect  specific  operational  events  that  must  occur. 

e.g,  ship  relative  movements,  i.e.,  ships  must  be  in  specific  relative  positions  and  distances 
for  the  ThreadEx  to  be  successful. 

e.g.,  a  White  Cell  event  that  injects  operational  information. 

System  Conditions 

Conditions  at  which  any  system  need  to  be  operating  for  correct  data  capture. 

Information  Conditions 

Needed  information  conditions  for  correct  data  production. 

e.g.,  formats,  type  of  information  to  be  available,  informatin  load,  etc. 

System  Log  Measures 

Specific  measures  to  be  determined  from  electronic  system  logs. 

System  from  which  the  log  is  to  be  obtained. 

Format  Attribute:  System:  measure  description,  second  measure  description;... 

e.g.,  Timely:  GCCS:  elapsed  time  between  information  sent  and  received,  elapsed  time 
between  information  received  and  displayed. 

Chat  Log  Measures 

Specific  measures  to  be  determined  from  Chat  logs. 

Chat  Room  from  which  the  log  is  to  be  obtained. 

Format  Attribute:  Chat  Room:  measure  description;  second  measure  description;... 
e.g.,  Reach:  IRS  Analysts:  number  of  people  using  chat  as  a  function  of  situation. 

Efficiency:  IRS:  number  of  mouse  clicks  required  to  log  on. 

38 


Observer  Log  Measures 

Specific  measures  to  be  determined  from  SME-Observer  logged  event  information. 

Logger  who  is  to  record  the  information.  Format  Attribute:  Logger:  measure  description, 

second  measure  description;  e.g.,  Timely:  UAV  Observer:  elapsed  time  between  request 

for  imagery  and  target  obtained,  elapsed  time  between  target  obtained  and  image  chipped, 
elapsed  time  from  image  obtained  and  image  sent. 

SME  Observer  QuestionsSpecific  questions  to  be  answered  by  an  SME-Observer. 

Specify  the  SME. 

Format  Attribute:  SME:  question,  second  question;... 

e.g.,  Timely:  CIC  Observer:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  Observer:  Was  their  sufficient  information  to  provide  the  needed  situation 
assessment? 

Survey  Questions 

Specific  questions  to  be  answered  by  operator  survey. 

Specify  operators  to  be  surveyed. 

In  most  cases  the  SME  and  Operator  questions  will  be  the  same. 

Format  Attribute:  personnel:  question,  second  question.... 

e.g.,  Timely:  CIC  personnel:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  analysts:  Was  there  sufficient  information  to  provide  the  needed  situation 
assessment? 

Participant  Status 

Coordination  prior  to  each  event. 

There  must  be  coordination  between  the  parties  who  are  participating  in  the  ThreadEx  to 
determine  their  status  PRIOR  to  starting  the  ThreadEx. 

The  parties  could  be  at  a  single  location  or  on  multiple  platforms. 

The  TC  must  contact  all  parties  PRIOR  to  starting  the  ThreadEx  to  determine  their  status. 

Flow  will  this  be  done?  Typically  by  email  or  chat. 

The  response  or  format  by  each  participant  can  vary  but  must  be  agreed  upon  PRIOR  to  the 
ThreadEx,  i.e.,  "Go/No  Go"  by  system(s)  required  to  support  the  ThreadEx. 

Instructions  /  Procedures 

Instructions  for  personnel  so  that  they  can  help  direct  Thread  activities. 

What  are  you  going  to  do  step-by-step? 

Step  1.  Disconnect  widget  1  from  widget  2.  Step  2.  Step  3. 

Data  Collection 

Instructions 

Specific  instructions  for  the  ThreadEx  data  collector. 

1.  What  data  should  be  collected? 

e.g.,  A  survey  will  be  performed,  Observation  Logs  maintained,  Electronic  Logs  captured 
from  systems,  etc. 

2.  Where  will  the  data  be  collected  and  with  what  medium? 

3.  Who  will  collect  the  data? 

4.  When  will  the  data  be  collected? 

Reporting  Instructions 

Instructions  for  all  reporting  to  be  done  by  experiment  personnel. 

daily,  after  action,  quicklook,  etc. 

Event  Review  Comments 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  events  and  conditions. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Data  Review  Comments 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  data  to  be  captured  and  participant  directions. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Table  8.  Directions  for  events  and  data  planning  input  (ThreadEx). 
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9.3  Analysis  Planning  Directions 


Analysis  Planning  Directions 

Thread  identifier  using  specified  format. 

Status 

Planning  components  status. 

Choose  status  from  the  drop-down. 

Only  Reviewer  can  choose  Reviewed  or  Complete. 

Reviewed  =  review  has  been  conducted  and  modifications  are  needed. 

Modified  (Lead  or  SME  use  only)  =  modifications  have  been  made  in  response  to  reviewer's 
comments  and  another  review  is  needed. 

System  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  electronic  system  logs. 

Specify  the  system  log  from  which  data  will  be  obtained. 

Format  "Attribute:  System:  measure  description,  second  measure  description;", 
e.g.,  Timely:  GCCS:  elapsed  time  between  information  sent  and  received,  elapsed  time 

System  Analysis  Analysis  procedures  to  be  used  to  determine  the  measures  from  the  system  log  data. 

Procedures  e.g.,  average,  distribution  plot,  histogram,  max  and  min,  ratio,  etc. 


System  Analysis  Current  system  data  analysis  status. 

Status  Status  Needed  procedure  changes  Roadblocks. 


System  Analyst  Name  of  the  person  responsible  for  performing  system  data  analysis. 


Lnai Log 
Measures 

apecmc  purtmiiAiivt  measures  to  De  determined  rrom  mat  logs. 

Specify  the  Chat  Room  from  which  the  log  is  to  be  obtained. 

Format  "Attribute:  Chat  Room:  measure  description;  second  measure  description;  etc", 

e.g.,  Reach:  IRS  Analysts:  number  of  people  using  chat  as  a  function  of  situation. 

Efficiency:  IRS:  number  of  mouse  clicks  required  to  log  on. 

Chat  Analysis 
Procedures 

Analysis  procedures  to  be  used  to  determine  the  measures  from  the  Chat  log 
information. 

e.g.,  average,  distribution  plot,  histogram,  max  and  min,  ratio,  etc. 

Chat  Analysis 
Status 

Current  Chat  information  analysis  status. 

Status  Needed  procedure  changes  Roadblocks. 

Chat  Analyst 

Name  of  the  person  responsible  for  performing  chat  log  analysis. 

Observer  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  SME-Observer  logged  event 
information. 

Specify  the  logger  who  is  to  record  the  information. 

Format  "Attribute:  Logger:  measure  description,  second  measure  description;". 

e.g.,  Timely:  UAV  Observer:  elapsed  time  between  request  for  imagery  and  target  obtained, 
elapsed  time  between  target  obtained  and  image  chipped,  elapsed  time  from  image  obtained  and 
image  sent. 

Obs.  Measures 

Analysis 

Procedures 

Analysis  procedures  to  be  used  to  determine  the  measures  from  the  Chat  log 
information. 

e.g.,  average,  distribution  plot,  histogram,  max  and  min,  ratio,  etc. 

Obs.  Measures 
Anal.  Status 

Current  observer  measures  analysis  status. 

Status  Needed  procedure  changes  Roadblocks. 

Obs.  Measures 

Name  of  the  person  responsible  for  performing  observer  log  analysis. 

SME  Observer  Specific  questions  to  be  answered  by  an  SME-Observer  (SUBJECTIVE) 


Format  "Attribute:  SME:  question,  second  question;". 

e.g.,  Timely:  CIC  Observer:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  Observer:  Was  there  sufficient  information  to  provide  the  needed  situation 
assessment? 

SME  Questions 

Analysis 

Procedures 

Analysis  procedures  to  be  used  to  determine  Attribute  question  results  from  the  SME- 
Observer  answers  to  questions. 

e.g.,  scale,  average  over  observers,  etc. 

SME  Questions 
Anal.  Status 

Current  observer  questions  analysis  status. 

Status  Needed  procedure  changes  Roadblocks. 

SME  Questions 
Analyst 

Name  of  the  person  responsible  for  performing  observer  questions  analysis. 

40 


Survey 

Questions 

Specific  questions  to  be  answered  by  operator  survey. 

Specify  operators  to  be  surveyed. 

In  most  cases  the  SME  and  Operator  questions  will  be  the  same. 

Format  Attribute:  personnel:  question,  second  question.... 

e.g.,  Timely:  CIC  personnel:  Was  information  available  in  time  for  Commander's  briefing? 
Sufficient:  J3  analysts:  Was  there  sufficient  information  to  provide  the  needed  situation 
assessment? 

Survey  Quest. 

Analysis 

Procedures 

Analysis  procedures  to  be  used  to  determine  Attribute  question  results  from  operator 
answers  to  questions. 

e.g.,  scale,  average  over  observers,  etc. 

Survey  Anal. 

Current  operator  questions  analysis  status. 

Survey  Analyst  Name  of  the  person  responsible  for  performing  observer  questions  analysis. 


Aggregation 

Procedures 

Analysis  procedures  to  be  used  to  fuse  various  results. 

E.g.,  Several  measures  into  a  single  Attributes  metric. 

Objective  and  subjective  into  an  Attribute  result. 

MDPq  infn  an  MDF 

Thread  Context  Procedures  to  be  used  to  extract  the  context/situation  under  which  this  Thread's  data 
Procedures  were  captured. 


Objective- 

Question 

Each  Objective-Question  contributes  to  achieving  the  Objective. 

There  may  be  several  Objective  Questions  for  an  Objective. 

Only  a  single  question  is  allowed  for  each  Thread.  If  more  than  one  question  is  needed,  use 
multiple  threads. 

The  attributes  for  the  measures  to  be  obtained  must  be  in  the  question.  Sufficient  detail  is 
provide  in  the  question  to  guide  formulating  needed  measures. 

Objective- 

Question 

Analysis 

Procedure 

Analysis  procedures  to  produce  the  Objective-Question  answer. 

Yes/no  answer  only  not  acceptable,  must  contain  explanation,  including  context, 
e.g.,  Fusion  of  included  Attribute  measures/answers. 

Weighting  of  the  various  contributing  results. 

Purely  subjective  from  correlation  of  SME  and  operator  opinions. 

Obj. -Quest. 

Anal.  Status 

Current  Objective-Question  analysis  status. 

Status  Needed  procedure  changes  Roadblocks. 

Obj. -Quest. 
Analyst 

Name  of  the  person  responsible  for  performing  objective-question  analysis. 

Objective 

High-level  statement  of  an  operational  or  developmental  purpose. 

Only  a  single  sentence  allowed. 

All  Objectives  with  the  same  Objective  number  must  have  the  same  statement. 

Details  of  specifics  to  be  determined  go  in  the  Objective-Question. 

An  Objective  can  have  several  Objective-Questions. 

Objective 

Analysis 

Procedures 

Analysis  procedures  to  produce  the  Objective  status. 

e.g.,  Fusion  of  answers  from  included  Objective-Questions. 

Weighting  of  the  various  contributing  results. 

Purely  subjective  from  operator  opinions. 

Objective  Anal. 
Status 

Current  Objective-Question  analysis  status. 

Status  Needed  procedure  changes  Roadblocks. 

Objective 

Analyst 

Name  of  the  person  responsible  for  performing  objective-question  analysis. 

Attrib.  &  Meas. 

Procedures 

Review 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  attributes  and  measures  analysis  procedures. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Obj. -Quest.  & 
Obj.  Proced. 
Review 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  Objective  and  Objective-Question  analysis  procedures. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Table  9.  Directions  for  Analysis  planning  input. 
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9.4  Results  Input  Directions 


Results  Input  Directions 

Thread  # 

Thread  identifier  using  specified  format. 

Status 

Results  components  status. 

Choose  status  from  the  drop-down. 

Only  Reviewer  can  choose  Reviewed  or  Complete. 

Reviewed  =  review  has  been  conducted  and  modifications  are  needed. 

Modified  (Lead  or  SME  use  only)  =  modifications  have  been  made  in  response  to 
reviewer's  comments  and  another  review  is  needed. 

System  Log  Measures  Specific  QUANTITATIVE  measures  to  be  determined  from  electronic  system  logs. 

Specify  the  system  log  from  which  data  will  be  obtained. 


Format  "Attribute:  System:  measure  description,  second  measure  description;", 
e.g.,  Timely:  GCCS:  elapsed  time  between  information  sent  and  received,  elapsed  time 
between  information  received  and  displayed. 

System  Log  Measures 
Values 

Numerical  value  of  each  prescribed  measure  from  system  logs. 

Folow  the  analysis  plan  for  producing  these  measures. 

Format  Measure:  Result. 

Chat  Log  Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  Chat  logs. 

Specify  the  Chat  Room  from  which  the  log  is  to  be  obtained. 

Format  "Attribute:  Chat  Room:  measure  description;  second  measure  description;  etc", 

e.g.,  Reach:  IRS  Analysts:  number  of  people  using  chat  as  a  function  of  situation. 
Efficiency:  IRS:  number  of  mouse  clicks  required  to  log  on. 

Chat  Log  Measures 
Values 

Numerical  value  of  each  prescribed  measure  from  Chat  logs. 

Folow  the  analysis  plan  for  producing  these  measures. 

Format  Measure:  Result. 

Observer  Log 
Measures 

Specific  QUANTITATIVE  measures  to  be  determined  from  SME-Observer  logged 
event  information. 

Specify  the  logger  who  is  to  record  the  information. 

Format  "Attribute:  Logger:  measure  description,  second  measure  description;". 

e.g.,  Timely:  UAV  Observer:  elapsed  time  between  request  for  imagery  and  target 
obtained,  elapsed  time  between  target  obtained  and  image  chipped,  elapsed  time  from 
image  obtained  and  image  sent. 

Observer  Log  Measures 
Values 

Numerical  value  of  each  prescribed  measure  from  Observer  logs. 

Folow  the  analysis  plan  for  producing  these  measures. 

Format  Measure:  Result. 

SME  Observer 
Questions 

Specific  questions  to  be  answered  by  an  SME-Observer  (SUBJECTIVE). 

Specify  the  SME. 

Format  "Attribute:  SME:  question,  second  question;". 

e.g.,  Timely:  CIC  Observer:  Was  information  available  in  time  for  Commander's 
briefing?  Sufficient:  J3  Observer:  Was  there  sufficient  information  to  provide  the  needed 
situation  assessment? 

SME  Observer  Answers 

Analysis  processed  answers  to  each  of  the  SME  answered  questions. 

Follow  the  analysis  plan  for  this  information. 

Include  individual  answers  and  summaries  as  needed  for  full  understanding  of  the 
results. 

Provide  a  result  for  each  Attribute. 

Survey  Questions 

Specific  questions  to  be  answered  by  operator  survey. 

Specify  operators  to  be  surveyed. 

In  most  cases  the  SME  and  Operator  questions  will  be  the  same. 

Format  Attribute:  personnel:  question,  second  question.... 

e.g.,  Timely:  CIC  personnel:  Was  information  available  in  time  for  Commander's 
briefing?  Sufficient:  J3  analysts:  Was  there  sufficient  information  to  provide  the  needed 
situation  assessment? 
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Survey  Answers 

Analysis  processed  answers  to  each  of  the  SME  answered  questions. 

Follow  the  analysis  plan  for  this  information. 

Include  individual  answers  and  summaries  as  needed  for  full  understanding  of  the 
results. 

Provide  a  result  for  each  Attribute. 

Thread  Context 

Pertinent  context/situation  under  which  data  and  information  for  this  Thread 
were  gathered. 

Include  any  context  that  influences  the  results. 

e.g.,  system  malfunction,  needed  information  absent,  personnel  untrained,  etc. 

Objective-Question 

Each  Objective-Question  contributes  to  achieving  the  Objective. 

There  may  be  several  Objective  Questions  for  an  Objective. 

Only  a  single  question  is  allowed  for  each  Thread.  If  more  than  one  question  is  needed, 
use  multiple  threads. 

The  attributes  for  the  measures  to  be  obtained  must  be  in  the  question.  Sufficient 
detail  is  provide  in  the  question  to  guide  formulating  needed  measures. 

Objective-Question 

Answer 

Statement  answer  to  the  Objective-Question. 

Not  yes/no,  and  any  desired  elaboration. 

Some  elaboration  will  be  contained  in  the  below  Context  Impact. 

Context  Impact 

Impact  of  context/situation  on  the  results 

e.g.,  validity,  Applicatility  limitations,  need  for  further  study,  etc. 

Measures  Results  Review  Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  measures  and  question  answers. 


Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Objective-Question  and 
Context  Results  Review 

Reviewer  guidance  for  needed  input  modifications. 

This  review  is  for  Context,  Objective-Question  results,  and  Objective  results. 

Choose  the  appropriate  status  drop-down  after  review  or  modifications. 

Table  10.  Directions  for  results  input. 
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10.  Example  Experiment  Thread 


This  planning  example  presents  (in  brief)  the  full  holistic  thought  process  that  leads  to  specific 
experiment  components. 

Planning  components  are  not  presented  for  Analysis. 

Overarching  purpose 

Operational  activity  supported  -  Re-Tasking  of  an  en-route  strike  aircraft.  Thus 

Objective:  Provide  and  distribute  information  for  dynamic 

re-tasking  of  en-route  strike  aircraft. 

General  planning  considerations 

The  system  RT-101  has  been  developed  and  is  to  be  tested.  The  system  is  to  provide  information 
to  both  AOC/MOC  decision-makers  and  to  the  aircraft.  The  overarching  purpose  of  the  test  is  to 
determine  if  the  system: 

•  Supports  decision-making  by  ground  units. 

•  Provides  both  tasking  and  targeting  information  in  the  aircraft  that  allow  the  mission  to 
be  prosecuted  successfully. 

•  Supports  TST  re-tasking  timelines. 

Thread  (or  multi-Thread)  formulation  considerations: 

What  is  to  be  learned: 

•  Whether  the  aircraft  information  presentation  supports  target  engagement. 

•  Whether  the  information  supports  TST  timelines. 

•  Whether  existing  comms  are  sufficient  to  support  the  system. 

The  associated  Objective-Questions  are: 

Thread- 1  Is  RT-101  capable  of  providing  sufficient,  accurate,  and  clear 

targeting  information  to  aircraft  for  TST  engagement? 

Thread-2  Does  RT-101  provide  timely  TST  information  for  re-tasking  decision¬ 
making  by  the  AOC  and  WOC  and  prosecution  by  aircraft  in  flight? 

Thread-3  What  bandwidth  is  required  to  support  RT-101? 

Attributes 

Thread- 1  has  an  MOE  (capable)  and  three  MOPs  so  a  roll-up  could  be  required.  The  roll-up  is 
too  hard  so  capable  will  be  asked  as  a  subjective  question.  The  other  threads  have  only  MOPs. 
Thread- 1  Capable,  Sufficient,  Accurate,  Clear 

Thread-2  Timely 

Thread-3  Capacity 

Measures 

Thread- 1  Accurate:  RT-101  display  capture:  TLE  on  display  (m); 
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Thread-2 

Sufficient:  RT-101  display  capture:  ratio  of  number  missing  to 
number  of  required  elements,  number  of  required  elements  unclear. 

Timely:  RT-101  log:  time  lapse  (min)  for  -  receipt  of  TST 
information  at  AOC  -  sending  prosecution  request  to  WOC 
-  re-tasking  sent  to  aircraft  -  re-tasking  receipt  at  aircraft  - 
aircraft  response  to  request  -  target  engagement. 

Thread-3 

Capacity:  link  sniffer:  RT-101  bandwidth  usage. 

Questions 

Thread- 1 

Capable:  AOC  observer:  WOC  operator:  Pilot:  Is  RT-101 
capable  of  supporting  TST  re-tasking?  Sufficient:  AOC 

Observer:  WOC  operator:  Is  RT-101  information  sufficient 
for  making  re-tasking  decisions?  Accurate:  Pilot:  Is  RT-101 
information  accurate  enough  to  engage  the  target?  Clear: 

Pilot:  Is  RT-101  display  clear  for  target  engagement? 

Thread-2 

Thread-3 

None 

None 

Type  of  test 

•  Live  test,  aircraft  in  flight. 

•  Both  AOC  COD/DOT  and  Wing  engaged  in  the  re-tasking  decision. 

•  Tipper  injected  by  the  White  Cell,  no  live  ISR. 

•  Target  information  pre-formulated  and  input  to  system,  by  system  operator,  when  tipper 
received. 

Conditions  (for  all  3  Threads) 

Operational:  Strike  aircraft  in  flight,  WOC  decision  unit  functioning, 


AOC  decision  unit  functioning. 

System: 

RT-101  in  operation  at  each  node.  Standard  comms  links. 

Information: 

Full  information  on  target,  including  imagery,  injected  not  retrieved 
from  MIDB.  Assume  mensuration  not  needed.  Injected  ATO  of 
sufficient  detail  (Wing  assignments)  to  make  aircraft  re-tasking  decision. 
WOC  inject  of  aircraft  assignments  and  load-out.  CDE  not  needed. 

ThreadEx  Procedures: 

Aircraft  assigned  strike  mission  through  ATO.  TST  information 
forwarded  from  White  Cell  to  RT101  operator.  Operator  injects 
TST  information.  SME  observers  on  station  at  WOC  and  AOC 
to  capture  information  events  on  RT-101. 

Data  Collection  Instructions: 

SME  observers  download  RT-101  logs  at  the  end  of  each  re-tasking 
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mission.  SME  observers  direct  pilot  and  AOC  and  WOC  operators  to 
take  survey  at  the  end  of  each  re-tasking  mission.  SME  observer  file 
event  log  at  the  end  of  each  experiment  day. 
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